Camera abnormality detecting device and program

ABSTRACT

A camera abnormality detecting device according to an embodiment includes an acquiring section and a notifying section. The acquiring section acquires, from an image photographed by a camera provided in a conveyance body to be capable of photographing an inside of a basket placed on a basket receiving section of the conveyance body, an image of a region where an upper edge of the basket is surmised to be reflected. The notifying section notifies, based on the image acquired by the acquiring section, whether an angle of view of the camera is blocked.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2018-196726, filed in Oct. 18, 2018, theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a camera abnormalitydetecting device and a program for causing a computer to function as thecamera abnormality detecting device.

BACKGROUND

In recent years, a cart POS (Point Of Sales) system has been proposedfor a small store equipped with shopping carts. In the followingexplanation, the shopping cart is referred to as cart. In the system, aninformation terminal including a touch panel and a scanner is providedin the cart. If placing a purchased commodity on the cart, a customerusing the cart causes the scanner to read an identification code of thepurchased commodity and performs commodity registration. Since thecustomer himself or herself performs the commodity registration whileperforming shopping in this way, it is possible to reduce congestion inan accounting area, at a so-called register. Further, if a self-typeaccounting machine is used, the customer can perform registration toaccounting of purchased commodities thoroughly by himself or herself.

On the other hand, in the case of the cart POS system, if a customerputs a purchased commodity in a basket placed on a basket receivingsection of the cart in a selling area where commodities are displayed,the customer performs commodity registration by himself or herself. Astore clerk cannot easily monitor the customer. Therefore, it isconceivable to monitor the customer by setting a camera in the cart tobe able to photograph the basket in the basket receiving section.However, even if the camera is set in the cart, the camera cannotphotograph the basket if an angle of view of the camera is blocked.Therefore, there is a concern that a monitoring effect by the cameradecreases.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overall configuration diagram of a cart POS systemaccording to a first embodiment;

FIG. 2 is a block diagram illustrating a main part circuit configurationof an information terminal in the first embodiment;

FIG. 3 is a schematic diagram illustrating a main work area provided ina main memory illustrated in FIG. 2;

FIG. 4 is a perspective view illustrating an example of a cart in whichthe information terminal is provided in the first embodiment;

FIG. 5 is a top view of a basket placed on the cart in the firstembodiment;

FIG. 6 is a schematic diagram for explaining a basic image in the firstembodiment;

FIG. 7 is a flowchart illustrating a main procedure of processingexecuted by a processor of the information terminal according to acomputer program in the first embodiment;

FIG. 8 is a schematic diagram for explaining a basic image in a secondembodiment;

FIG. 9 is a flowchart illustrating a main procedure of processingexecuted by a processor of an information terminal according to acomputer program in the second embodiment;

FIG. 10 is a schematic diagram illustrating a first state in which acommodity is stored in a basket in the second embodiment; and

FIG. 11 is a schematic diagram illustrating a second state in whichcommodities are stored in the basket in the second embodiment.

DETAILED DESCRIPTION

An object of embodiments is to provide a camera abnormality detectingdevice and a program for the camera abnormality detecting device thatcan detect, on a real-time basis, a situation in which an angle of viewof a camera is blocked and can improve a monitoring effect by thecamera.

According to an embodiment, a camera abnormality detecting deviceincludes an acquiring section and a notifying section. The acquiringsection acquires, from an image photographed by a camera provided in aconveyance body to be capable of photographing an inside of a basketplaced on a basket receiving section of the conveyance body, an image ofa region where an upper edge of the basket is surmised to be reflected.The notifying section notifies, based on the image acquired by theacquiring section, whether an angle of view of the camera is blocked.

Embodiments of a camera abnormality detecting device that can detect, ona real-time basis, a situation in which an angle of view of a camera isblocked and can improve a monitoring effect by the camera are explainedbelow with reference to the drawings. In the embodiments, a situation inwhich an angle of view of a camera attached to a cart used in afull-self-type cart POS system is blocked is detected as abnormality.

First Embodiment

FIG. 1 is an overall configuration diagram of a cart POS system 1according to a first embodiment. The cart POS system 1 includes a server10, accounting machines 20, information terminals 30, and an attendantterminal 40. The information terminals 30 are respectively provided in aplurality of carts C. The cart C is an example of a conveyance body thatconveys purchased commodities of a customer, who is a user of the cartC, put in a basket.

The server 10, the accounting machines 20, and the attendant terminal 40are connected to a network 50 such as a LAN (Local Area Network). Anaccess point 51 is also connected to the network 50. The informationterminals 30 include wireless units. The information terminals 30perform wireless communication with the access point 51. The accesspoint 51 relays communication between the devices connected to thenetwork 50, that is, the server 10, the accounting machines 20, and theattendant terminal 40 and the information terminals 30. Only one accesspoint 51 is illustrated in FIG. 1. However, two or more access points 51may be present depending on the size and the like of a store.Consequently, the information terminals 30 can perform datacommunication with the server 10, the accounting machines 20, and theattendant terminal 40 via the access point 51 and the network 50.

The information terminal 30 is a device with which a customer performsdata input related to registration of purchased commodities by himselfor herself. The accounting machine 20 is a device with which thecustomer performs accounting of the purchased commodities by himself orherself. The cart POS system 1 realizes a full-self system in which thecustomer performs the registration to the accounting of the purchasedcommodities by himself or herself. A plurality of accounting machines 20may be set in one store. In FIG. 1, the cart POS system 1 in which threeaccounting machines 20 are set in one store is illustrated.

The attendant terminal 40 is provided in an attendant counter where awatchman, a so-called attendant is stationed. The attendant terminal 40includes a display device for displaying states of the informationterminals 30 and the accounting machines 20. The attendant monitors, ona real-time basis, the states of the information terminals 30 and theaccounting machines 20 from information displayed on the display device.

The server 10 has a function of a virtual POS terminal between theinformation terminals 30 and the accounting machines 20. That is, theserver 10 creates list data of purchased commodities from data input inthe information terminal 30. The server 10 specifies a customer who usesthe accounting machine 20 and transmits list data of purchasedcommodities of the customer to the accounting machine 20.

FIG. 2 is a block diagram illustrating a main part circuit configurationof the information terminal 30. The information terminal 30 includes aprocessor 31, a main memory 32, an auxiliary storage device 33, awireless unit 34, a touch panel 35, a scanner 36, a reader 37, a camera38, and a system transmission line 39. The system transmission line 39includes an address bus, a data bus, and a control signal line. In theinformation terminal 30, the processor 31, the main memory 32, theauxiliary storage device 33, the wireless unit 34, the touch panel 35,the scanner 36, the reader 37, and the camera 38 are connected to thesystem transmission line 39. In the information terminal 30, a computeris configured by the processor 31, the main memory 32, and the auxiliarystorage device 33 and the system transmission line 39 that connectsthese devices.

The processor 31 is equivalent to a central part of the computer. Theprocessor 31 controls the sections according to an operating system orapplication programs in order to realize various functions of theinformation terminal 30. The processor 31 is, for example, a CPU(Central Processing Unit).

The main memory 32 is equivalent to a main storage portion of thecomputer. The main memory 32 includes a nonvolatile memory region and avolatile memory region. The main memory 32 stores the operating systemor the application programs in the nonvolatile memory region. The mainmemory 32 sometimes stores, in the nonvolatile or volatile memoryregion, data necessary for the processor 31 in executing processing forcontrolling the sections. The main memory 32 uses the volatile memoryregion as a work area where data is rewritten as appropriate by theprocessor 31. As illustrated in FIG. 3 in particular, the main memory 32includes a first image memory M1, a second image memory M2, and a memoryM3 of a warning counter n as the work area. The nonvolatile memoryregion is, for example, a ROM (Read Only Memory). The volatile memoryregion is, for example, a RAM (Random Access Memory).

The auxiliary storage device 33 is equivalent to an auxiliary storageportion of the computer. For example, an EEPROM (Electric ErasableProgrammable Read-Only Memory), a HDD (Hard Disk Drive), or an SSD(Solid State Drive) can be the auxiliary storage device 33. Theauxiliary storage device 33 saves data used by the processor 31 inperforming various kinds of processing, data created by the processingin the processor 31, or the like. The auxiliary storage device 33sometimes stores the application programs.

The wireless unit 34 controls wireless communication performed betweenthe information terminal 30 and the access point 51 according to awireless communication protocol.

The touch panel 35 is a device functioning as both of an input deviceand a display device of the information terminal 30. The touch panel 35detects a touch position on a displayed image and outputs informationconcerning the touch position to the processor 31.

The scanner 36 reads a code symbol such as a barcode or atwo-dimensional data code attached to a commodity. A code symbolrepresenting an identification code of the commodity is attached to thecommodity. The scanner 36 outputs data of the read code symbol to theprocessor 31. The scanner 36 may be a type for reading the code symbolwith scanning of a laser beam or may be a type for reading the codesymbol from an image captured by an imaging device.

The reader 37 reads data recorded in a recording medium and outputs theread data to the processor 31. If the recording medium is a magneticcard, the reader 37 is a magnetic card reader. If the recording mediumis a contact IC card, the reader 37 is an IC card reader. In the case ofa recording medium in which RFID (Radio Frequency Identification) isused such as a noncontact IC card or a smartphone, an RFID reader isused as the reader 37.

The camera 38 is provided in the cart C to be able to photograph abasket placed on the basket receiving section of the cart C from above.The camera 38 is a camera for monitoring whether a customer, who is auser of the cart C, correctly puts purchased commodities in the basket.

In the information terminal 30 including the circuit componentsexplained above, the processor 31, the main memory 32, the auxiliarystorage device 33, the wireless unit 34, and the touch panel 35configure a tablet terminal TM. The information terminal 30 isconfigured by electrically connecting the scanner 36, the reader 37, andthe camera 38 to the tablet terminal TM via the system transmission line39.

FIG. 4 is a perspective view illustrating an example of the cart C inwhich the information terminal 30 is provided. The cart C includes acaster section C1 for movement, a handle frame section C2, and a basketreceiving section C3. The caster section C1 includes four wheels C11 forsmoothly moving the cart C on a floor surface. The caster section C1includes a receiving section C12 for placing a large baggage that cannotbe put in a basket 60. The handle frame section C2 includes a pair ofvertical frames C21, C21 erected on a rear wheel side of the castersection C1 and a handlebar C22 that couples upper ends of the verticalframes C21, C21. The basket receiving section C3 is present in the frontof a halfway part of the handle frame section C2. In the cart C, thebasket 60 provided in the store can be placed on the basket receivingsection C3. The basket 60 is a basket for storing commodities.

The scanner 36 is present in a halfway part of the handlebar C22. Thescanner 36 is attached to the handlebar C22 such that a reading windowis located on the near side. The near side is a side where a customerholding the handlebar C22 and pushing the cart C stands.

A pole C4 is attached to one vertical frame C21. The distal end of thepole C4 is located above the handlebar C22. The tablet terminal TM isattached to the distal end portion of the pole C4 with a screen of thetouch panel 35 facing the near side. The reader 37 is attached to thetablet terminal TM such that a card slit is located on the near side. InFIG. 4, the reader 37 is a magnetic card reader. The camera 38 isattached to a halfway part of the pole C4 to image the entire basket 60placed on the basket receiving section C3 from above.

A battery BT is attached between the vertical frames C21, C21 on thelower end side of the handle frame section C2. The battery BT functionsas a driving power supply for the tablet terminal TM, the scanner 36,the reader 37, and the camera 38.

FIG. 5 is a top view of the basket 60. A storing section 63 opened inthe upper surface is formed on the inside of the basket 60 by erectingsidewalls 62A, 62B, 62C, and 62D of four surfaces from edges of arectangular bottom wall 61 to slightly incline to the outer side. In thebasket 60, a pair of C-shaped handles 64A and 64B are attached to upperend portions 63A, 63B, 63C, and 63D of the sidewalls 62A, 62B, 62C, and62D. One handle 64A is laid closer to the sidewall 62D between the upperend portions 63A and 63C of a pair of sidewalls 62A and 62C erected fromedges in the longitudinal direction of the bottom wall 61 and isturnably supported with respect to the upper end portions 63A and 63C.If the handle 64A turns to the side of the sidewall 62D, the handle 64Acomes into contact with the upper end portion 63D of the sidewall 62D.The handle 64A and the upper end portions 63A, 63D, and 63C overlap. Theother handle 64B is laid closer to the sidewall 62B between the upperend portion 63A and the upper end portion 63C in the same manner and isturnably supported with respect to the upper end portions 63A and 63C.If the handle 64B turns to the side of the sidewall 62B, the handle 64Bcomes into contact with the upper end portion 63B of the sidewall 62B.The handle 64B and the upper end portions 63A, 63B, and 63C overlap.

As explained above, the camera 38 provided in the cart C photographs thebasket 60 placed on the basket receiving section C3 from above. Thebasket 60 is placed on the basket receiving section C3 such that thelongitudinal direction of the bottom wall 61 is parallel to a directionin which the cart C moves if a customer pushes the cart C. Therefore, ifan empty basket 60 is placed on the basket receiving section C3, imagesof the basket 60 photographed by the camera 38 are substantially thesame images, although the position of the basket 60 sometimes slightlydeviates in the longitudinal direction of the basket 60. On the otherhand, if commodities are stored in the storing section 63 of the basket60, images of the storing section 63 are different for each of thebaskets 60.

Therefore, in the first embodiment, as illustrated in FIG. 6, an imageof the upper end portions 63A, 63B, 63C, and 63D of the sidewalls 62A,62B, 62C, and 62D at the time when the basket 60 is viewed from aboveand the pair of handles 64A and 64B laid on the upper end portions 63A,63B, 63C, and 63D is set as a basic image P1 and stored in the auxiliarystorage device 33. The information terminal 30 segments, from the imagesphotographed by the camera 38, a portion where the upper edge of thebasket 60, that is, the upper end portions 63A, 63B, 63C, and 63D aresurmised to be reflected and compares the portion with the basic imageP1. If a matching rate is smaller than a first threshold A, theinformation terminal 30 determines that an angle of view of the camera38 is blocked and notifies the customer or the attendant. Theinformation terminal 30 has a function of a camera abnormality detectingdevice.

Operation in the first embodiment explained above is specificallyexplained below.

First, a customer who performs shopping using the cart C places theempty basket 60 on the basket receiving section C3. The customer touchesthe touch panel 35. If the touch panel 35 is touched, the processor 31in an idle state starts. The processor 31 starts information processingof a procedure illustrated in a flowchart of FIG. 7.

FIG. 7 is a flowchart illustrating a main procedure of processingexecuted by the processor 31 of the information terminal 30 according toa computer program. The computer program is stored in the main memory 32or the auxiliary storage device 33. Content of the processingillustrated in the flowchart of FIG. 7 and explained below is anexample. A procedure of the processing is not particularly limited ifthe same result can be obtained.

In Act 1, the processor 31 causes the touch panel 35 to display aregistration screen. The registration screen is, for example, a screenfor displaying names, prices, and the like of commodities registered bythe information terminal 30. The registration screen includes an imageof an accounting button with which the customer instructs accounting ofpurchased commodities. In Act 2, the processor 31 reads out the basicimage P1 from the auxiliary storage device 33 and stores the basic imageP1 in a second image memory M2. In Act 3, the processor 31 starts thecamera 38. In Act 4, the processor 31 resets the warning counter n ofthe warning counter memory M3 to “0”. The order of the processing inActs 1 to 4 is not limited to this and may be changed as appropriate.The processor 31 only has to execute the processing in Acts 1 to 4irrespective of order.

After ending the processing in Acts 1 to 4, the processor 31 proceeds toAct 5. In Act 5, the processor 31 confirms whether an identificationcode of a commodity is acquired. If an identification code is notacquired, the processor 31 determines NO in Act 5 and proceeds to Act 6.In Act 6, the processor 31 confirms whether the accounting button istouched. If the accounting button is not touched, the processor 31determines NO in Act 6 and proceeds to Act 9. The processor 31 executesprocessing in Acts 9 to 19 explained below. After ending the processingin Acts 9 to 19, the processor 31 returns to Act 5.

In Acts 5 and 6, the processor 31 waits for an identification code of acommodity to be acquired or the accounting button to be touched. Whilewaiting, the processor 31 repeatedly executes the processing in Acts 9to 19.

The customer starts shopping after confirming that the registrationscreen is displayed on the touch panel 35. If the customer finds acommodity that the customer desires to purchase, the customer causes thescanner 36 to read a code symbol such as a barcode attached to thecommodity and then puts the commodity in the basket 60. The code symbolincludes an identification code of the commodity to which the codesymbol is attached. If a code symbol is not attached to the commodity,the customer operates the touch panel 35 to cause the touch panel 35 todisplay a list of commodities without code symbols and selects, out ofthe list, a commodity that the customer desires to purchase. If thecommodity is selected, an identification code of the commodity is input.Therefore, the customer puts the commodity in the basket 60.

In the waiting state in Acts 5 and 6, if acquiring the identificationcode of the commodity via the scanner 36 or the touch panel 35, theprocessor 31 determines YES in Act 5 and proceeds to Act 7. In Act 7,the processor 31 performs registration processing for purchasedcommodities.

The processor 31 controls the wireless unit 34 to transmit commodityregistration data to the server 10. According to the control, thewireless unit 34 transmits the commodity registration data to the server10. The commodity registration data includes the acquired identificationcode of the commodity. The commodity registration data also includes aterminal ID set for each of terminals in order to individually identifythe information terminals 30. The server 10 receiving the commodityregistration data creates list data of purchased commodities for each ofterminals ID of the information terminals 30. The server 10 transmitscommodity data such as commodity names and prices of the purchasedcommodities to the information terminal 30. The processor 31 of theinformation terminal 30 updates the registration screen such that thecommodity data received from the server 10 is displayed on theregistration screen.

If the commodity put in the basket 60 is returned, first, the customeroperates the touch panel 35 to set a commodity return mode.Subsequently, the customer inputs the identification code of thecommodity to the information terminal 30 in the same manner as duringthe purchase and thereafter returns the commodity to an original shelf.At this time, the processor 31 performs commodity return processing forpurchased commodities. The processor 31 controls the wireless unit 34 totransmit commodity return data to the server 10. According to thecontrol, the wireless unit 34 transmits the commodity return data to theserver 10. The commodity return data includes the acquiredidentification code of the commodity. The commodity return data alsoincludes the terminal ID of the information terminal 30. The server 10receiving the commodity return data deletes data of the returnedcommodity from list data for each of terminal IDs. The server 10transmits commodity data such as a commodity name and a price of thereturned commodity to the information terminal 30. The processor 31 ofthe information terminal 30 updates the registration screen such thatthe commodity data received from the server 10 is deleted from theregistration screen.

After ending the registration processing for the purchased commodity orthe commodity return processing for the returned commodity, theprocessor 31 proceeds to Act 6. The processor 31 confirms whether theaccounting button is touched. If the accounting button is not touched,the processor 31 executes the processing in Acts 9 to 19. After endingthe processing in Acts 9 to 19, the processor 31 returns to Act 5. Theprocessor 31 returns to the waiting state in Acts 5 and 6.

The customer ending the shopping touches the accounting button displayedon the registration screen. In the waiting state in Acts 5 and 6, ifdetecting that the accounting button is touched via the touch panel 35,the processor 31 determines YES in Act 6 and proceeds to Act 8. In Act8, the processor 31 performs accounting processing.

The processor 31 controls the wireless unit 34 to transmit accountinginstruction data to the server 10. According to the control, thewireless unit 34 transmits the accounting instruction data to the server10. The accounting instruction data includes the terminal ID of theinformation terminal 30. The server 10 receiving the accountinginstruction data selects the accounting machine 20 that performsaccounting. For example, the server 10 selects the accounting machine 20not in use. The server 10 transmits, to the information terminal 30,data for informing that the accounting is performed by the accountingmachine 20. The processor 31 of the information terminal 30 causes,based on the data received from the server 10, the touch panel 35 todisplay a screen for guiding the customer to the accounting machine 20.

After ending the accounting processing, the processor 31 ends theinformation processing of the procedure illustrated in the flowchart.Thereafter, if a time in which no operation input is performed exceeds afixed time, the processor 31 changes to the idle state.

The processing in Acts 9 to 19 executed by the processor 31 in thewaiting state in Acts 5 and 6 is specifically explained.

In Act 9, the processor 31 takes in a frame image photographed by thecamera 38 and stores the image in the first image memory M1. In Act 10,the processor 31 analyzes the image stored in the first image memory M1and segments a portion where the upper edge of the basket 60, that is,the upper end portions 63A, 63B, 63C, and 63D are surmised to bereflected. The processor 31 compares the image segmented in Act 11 andthe basic image P1 stored in the second image memory M2. For example,the processor 31 collates, for each of pixels, luminance, colorcomponents, and the like of the segmented image and the basic image P1and calculates, for all the pixels, a ratio of matching pixels, that is,a matching rate.

In Act 12, the processor 31 determines whether the matching rate issmaller than a first threshold A. The first threshold A is optional. Forexample, if a commodity having a large entire length is stood againstone sidewall and stored in the basket 60, the upper end portion of thesidewall is sometimes hidden by a part of the commodity if the basket 60is viewed from above. The matching rate decreases if the upper endportion of the sidewall is hidden by the commodity. However, the angleof view of the camera 38 is not blocked. On the other hand, if thecommodity having the large entire length is stood against the front of alens of the camera 38 and most of the angle of view of the camera 38 isblocked, the matching rate greatly decreases. If the customerintentionally covers the lens with cloth or the like, the matching ratealso greatly decreases. The first threshold A is a value for identifyingwhether the angle of view of the camera 38 is blocked. An administratorof the system sets the first threshold A considering the event explainedabove.

If the matching rate is equal to or larger than the first threshold A,the processor 31 determines NO in Act 12 and proceeds to Act 13. In Act13, the processor 31 confirms whether the warning counter n is “1” ormore. If the warning counter n is not “1” or more, that is, if a warningis not generated, the processor 31 determines NO in Act 13 and returnsto Act 5.

If the matching rate is smaller than the first threshold A, in Act 14,the processor 31 counts up the warning counter n by “1”. In Act 15, theprocessor 31 determines whether the warning counter n exceeds a thirdthreshold N1.

In this embodiment, if the matching rate is smaller than the firstthreshold A, the processor 31 warns the customer, who is the user of theinformation terminal 30, that the angle of view of the camera 38 isblocked. However, if a fixed time T elapses while a situation in whichthe matching rate is smaller than the first threshold A is not improved,the processor 31 notifies the attendant that the angle of view of thecamera 38 is blocked. The third threshold N1 is a count value until thewarning counter n reaches the fixed time T. It is optional to determineto which degree of time the fixed time T is set.

If the warning counter n does not exceed the third threshold N1, theprocessor 31 determines NO in Act 15 and proceeds to Act 16. In Act 16,the processor 31 performs a first warning. The first warning is, forexample, processing for causing the touch panel 35 of the informationterminal 30 to display a message for notifying that the angle of view ofthe camera 38 is blocked. As the first warning, if the informationterminal 30 includes a buzzer, the buzzer may be sounded together withthe display of the message. After performing the first warning, theprocessor 31 returns to Act 5.

If the warning counter n exceeds the third threshold N1, the processor31 determines YES in Act 15 and proceeds to Act 17. In Act 17, theprocessor 31 performs a second warning. The second warning includes, forexample, in addition to the processing of the first warning, processingfor transmitting a warning signal from the wireless unit 34 to theserver 10. The warning signal is transmitted from the wireless unit 34to the server 10 by the processing of the second warning. The warningsignal includes an ID of the information terminal 30. The server 10controls the attendant terminal 40 such that a message for notifying theattendant that the angle of view of the camera 38 is blocked in theinformation terminal 30 identified by the ID is displayed on the displaydevice of the attendant terminal 40. After performing the secondwarning, the processor 31 returns to Act 5.

If the matching rate is equal to or larger than the first threshold Aand the warning counter n is “1” or more, the processor 31 determinesYES in Act 13 and proceeds to Act 18. In Act 18, the processor 31performs warning release. The first warning is released by the warningrelease. The message for the warning displayed on the touch panel 35 iserased. If the second warning is performed, the second warning is alsoreleased. The same message displayed on the display device of theattendant terminal 40 is erased.

In Act 19, the processor 31 resets the warning counter n to “0”.Thereafter, the processor 31 returns to Act 5.

The computer including the processor 31 as the central part configuresan acquiring section through the processing in Acts 9 and 10. Thecomputer configures a notifying section through the processing in Acts11, 12, and 14 to 17.

In this way, in the first embodiment, the processor 31 acquires, from animage photographed by the camera 38 provided in the cart C to be capableof photographing the entire basket 60 placed on the basket receivingsection C3 of the cart C, an image in which the upper edge of the basket60 is surmised to be reflected and stores the image in the first imagememory M1. On the other hand, an image of the upper end portions 63A,63B, 63C, and 63D of the sidewalls 62A, 62B, 62C, and 62D at the timewhen the basket 60 is viewed from above and the pair of handles 64A and64B laid on the upper end portions 63A, 63B, 63C, and 63D is stored inthe second image memory M2 as the basic image P1.

The processor 31 compares the image in the first image memory M1 and thebasic image P1 in the second image memory M2 and calculates a matchingrate of the image and the basic image P1. If the matching rate issmaller than the first threshold A, the processor 31 performs the firstwarning. Consequently, a message for warning that the angle of view ofthe camera 38 is blocked is displayed on the touch panel 35 of theinformation terminal 30. Therefore, the customer using the cart C canlearn from the message displayed on the touch panel 35 that the angle ofview of the camera 38 is blocked.

For example, most of the angle of view of the camera 38 is sometimesblocked because a commodity put in the basket 60 is located in front ofthe lens of the camera 38. In such a case, the customer confirming themessage moves the commodity in the basket 60 not to block the angle ofview of the camera 38. As a result, if the matching rate is equal to orlarger than the first threshold A, the message is erased. Consequently,the camera 38 can stably photograph a motion of the customer putting apurchased commodity in the basket 60 or a motion of the customer takingout a returned commodity from the basket 60. Therefore, it is possibleto improve a monitoring effect by the camera 38.

On the other hand, it is assumed that there is a customer whointentionally covers the lens or an opening of the basket 60 with clothor the like to prevent the basket 60 from being photographed by thecamera 38. Even in such a case, the message for warning that the angleof view of the camera 38 is blocked is displayed on the touch panel 35of the information terminal 30. However, the customer is considered toneglect the warning and leave the warning as it is. In such a case, inthis embodiment, if the waring counter n exceeds the third threshold N1,the processor 31 performs the second warning. Consequently, the messagefor notifying the attendant that the angle of view of the camera 38 isblocked in the information terminal 30 is displayed on the displaydevice of the attendant terminal 40.

Therefore, for example, the attendant can go to a selling area andadvice the customer to remove the cloth or the like covering the lens ofthe camera 38 or the opening of the basket 60. By widely informingcustomers that such an advice is performed, it is possible to prevent animproper act of intentionally covering the lens or the opening of thebasket 60 with cloth or the like. Therefore, it is possible to furtherimprove the monitoring effect by the camera 38.

Second Embodiment

A second embodiment is explained.

In the first embodiment, the image showing the upper edge of the basket60 placed on the basket receiving section C3 is set as the basic imageP1. In the second embodiment, in an initial stage, an image showing theempty basket 60 placed on the basket receiving section C3 is set as abasic image P2. Every time an event occurs in which a state of thestoring section 63 of the basket 60 changes, the basic image P2 isupdated to an image after the change. The portions explained withreference to FIGS. 1 to 5 in the first embodiment are common to thesecond embodiment. Therefore, the same portions are denoted by the samereference numerals and signs and explanation of the portions is omitted.

FIG. 8 is an example of the basic image P2 in the initial stage. Asillustrated in FIG. 8, the basic image P2 shows the bottom wall 61, thesidewalls 62A, 62B, 62C, and 62D of the four surfaces, the upper endportions 63A, 63B, 63C, and 63D of the sidewalls 62A, 62B, 62C, and 62D,and the C-shaped pair of handles 64A and 64B of the basket 60. Thestoring section 63 is empty.

The information terminal 30 stores the basic image P2 in the auxiliarystorage device 33. The information terminal 30 acquires, from an imagephotographed by the camera 38, an image including a region where thestoring section 63 of the basket 60 is surmised to be reflected. Theinformation terminal 30 recognizes an event in which a state of thestoring section 63 of the basket 60 changes. The information terminal 30recognizes an event in which a commodity is put in the storing section63 or an event in which a commodity is removed from the storing section63.

If recognizing such an event, the information terminal 30 compares theacquired image and the basic image P2 and calculates a degree of thechange. At this time, if the degree of the change is larger than asecond threshold B, the information terminal 30 updates the basic imageP2 to the acquired image. If recognizing, again, the event in which thestoring section 63 of the basket 60 changes, the information terminal 30compares an image acquired anew and the basic image P2 after the updateand calculates a degree of a change. At this time, if the degree of thechange is equal to or smaller than the second threshold B, theinformation terminal 30 determines that the angle of view of the camera38 is blocked and notifies the customer or the attendant. Theinformation terminal 30 has a function of a camera abnormality detectingdevice.

Operation in the second embodiment explained above is specificallyexplained below.

In the second embodiment, as in the first embodiment, a customer whoperforms shopping using the cart C places the empty basket 60 on thebasket receiving section C3. The customer touches the touch panel 35. Ifthe touch panel 35 is touched, the processor 31 in the idle statestarts. The processor 31 starts information processing of a procedureillustrated in FIG. 9.

FIG. 9 is a flowchart illustrating a main procedure of processingexecuted according to a computer program by the processor 31 of theinformation terminal 30. The computer program is stored in the mainmemory 32 or the auxiliary storage device 33. Content of processingillustrated in the flowchart of FIG. 9 and explained below is anexample. A procedure of the processing is not particularly limited ifthe same result can be obtained.

In Act 21, the processor 31 causes the touch panel 35 to display aregistration screen. The registration screen is common to the firstembodiment. In Act 22, the processor 31 reads out the basic image P2from the auxiliary storage device 33 and stores the basic image P2 inthe second image memory M2. In Act 23, the processor 31 starts thecamera 38. In Act 24, the processor 31 resets the warning counter n ofthe warning counter memory M3 to “0”. The order of the processing inActs 21 to 24 is not limited to this and may be changed as appropriate.The processor 31 only has to execute the processing in Acts 21 to 24irrespective of order.

After ending the processing in Acts 21 to 24, the processor 31 proceedsto Act 25. In Act 25, the processor 31 confirms whether anidentification code of a commodity is acquired. If an identificationcode is not acquired, the processor 31 determines NO in Act 25 andproceeds to Act 26. In Act 26, the processor 31 confirms whether theaccounting button of the registration screen is touched. If theaccounting button is not touched, the processor 31 determines NO in Act26 and returns to Act 25.

In Acts 25 and 26, the processor 31 waits for an identification code ofa commodity to be acquired or the accounting button to be touched. Inthis waiting state, if acquiring an identification code of a commodityvia the scanner 36 or the touch panel 35, the processor 31 determinesYES in Act 25 and proceeds to Act 27. In Act 27, as in the firstembodiment, the processor 31 performs the registration processing or thecommodity return processing for the purchased commodity.

After ending the registration processing or the commodity returnprocessing for the purchased commodity, the processor 31 proceeds to Act29. The processor 31 executes processing in Acts 29 to 40. Theprocessing in Acts 29 to 40 is explained below.

In the waiting state in Acts 25 and 26, if detecting that the accountingbutton is touched via the touch panel 35, the processor 31 determinesYES in Act 26 and proceeds to Act 28. In Act 28, the processor 31performs the accounting processing as in the first embodiment. Afterending the accounting processing, the processor 31 ends the informationprocessing of the procedure illustrated in the flowchart. Thereafter, ifa time in which no operation input is performed exceeds a fixed time,the processor 31 changes to the idle state.

The processing in Acts 29 to 40 executed by the processor 31 isspecifically explained.

After ending the registration processing or the commodity returnprocessing for the purchased commodity, in Act 29, the processor 31takes in a frame image photographed by the camera 38 and stores theimage in the first image memory M1. In Act 30, the processor 31 analyzesthe image stored in the first image memory M1 and segments a portionwhere an entire region including the storing section 63 of the basket 60is surmised to be reflected. The processor 31 compares the imagesegmented in Act 31 and the basic image P2 stored in the second imagememory M2. For example, the processor 31 collates, for each of pixels,luminance, color components, and the like of the segmented image and thebasic image P2 and calculates, for all the pixels, a ratio of changedpixels, that is, a change rate.

In Act 32, the processor 31 determines whether the change rate issmaller than a second threshold B. The second threshold B is optional.For example, if a commodity is put in the storing section 63 of thebasket 60, a change occurs in an image of the storing section 63 beforeand after the commodity is put in the storing section 63. On the otherhand, if the lens of the camera 38 is blocked, even if a commodity isput in the storing section 63, a change does not occur in a photographedimage before and after the commodity is put in the storing section 63.The same applies if a commodity is removed from the storing section 63.The second threshold B is a value for identifying whether the angle ofview of the camera 38 is blocked. The administrator of the system setsthe second threshold B considering the event explained above.

If the change rate is larger than the second threshold B, the processor31 determines YES in Act 32 and proceeds to Act 33. In Act 33, theprocessor 31 confirms whether the warning counter n is “1” or more. Ifthe warning counter n is not “1” or more, that is, if a warning is notgenerated, the processor 31 determines NO in Act 33 and proceeds to Act40.

On the other hand, if the change rate is equal to or smaller than thesecond threshold B, in Act 32, the processor 31 determines NO in Act 32and proceeds to Act 34. In Act 34, the processor 31 counts up thewarning counter n by “1”. In Act 35, the processor 31 determines whetherthe warning counter n exceeds a fourth threshold N2.

In this embodiment, if the change rate is equal to or smaller than thesecond threshold B, the processor 31 warns the user of the informationterminal 30 that the angle of view of the camera 38 is blocked. However,if a fixed number of times K of the registration processing or thecommodity return processing is carried out while a situation in whichthe change rate is equal to or smaller than the second threshold B isnot improved, the processor 31 notifies the attendant that the angle ofview of the camera 38 is blocked. The fourth threshold N2 is a valuesmaller than the fixed number of times K by “1”. It is optional todetermine to how many times the fixed number of times K is set.

If the warning counter n does not exceed the fourth threshold N2, theprocessor 31 determines NO in Act 35 and proceeds to Act 36. In Act 36,the processor 31 performs the first warning. Content of the firstwarning may be the same as the content of the first warning in the firstembodiment. After performing the first warning, the processor 31proceeds to Act 40.

If the warning counter n exceeds the fourth threshold N2, the processor31 determines YES in Act 35 and proceeds to Act 37. In Act 37, theprocessor 31 performs the second warning. Content of the second warningmay be the same as the content of the second warning in the firstembodiment. After performing the second warning, the processor 31proceeds to Act 40.

If the change rate is larger than the second threshold B and the warningcounter n is “1” or more, the processor 31 determines YES in Act 33 andproceeds to Act 38. In Act 38, the processor 31 performs the warningrelease as in the first embodiment. In Act 39, the processor 31 resetsthe warning counter n to “0”. Thereafter, the processor 31 proceeds toAct 40.

In this way, if the warning counter n is “0” in Act 33 or after endingthe processing in Act 36, 37, or 39, the processor 31 proceeds to Act40. In Act 40, the processor 31 stores the image segmented in theprocessing in Act 30 in the second image memory M2. The processor 31updates the basic image P2 stored in the second image memory M2 to animage in which the entire region including the storing section 63 of thebasket 60 is surmised to be reflected at that point in time.

If the basic image P2 stored in the second image memory M2 is changed toan image including the storing section 63 of the basket 60 after theregistration processing or the commodity return processing is performedin this way, the processor 31 returns to Act 25. Therefore, if theprocessor 31 acquires an identification code of a commodity anew andperforms the registration processing or the commodity return processingwith the identification code, the processor 31 executes the processingin Acts 29 to 40 again.

The computer including the processor 31 as the central part configures arecognizing section through the processing in Acts 25 and 27. Thecomputer configures an acquiring section through the processing in Acts29 and 30. The computer configures a notifying section through theprocessing in Acts 31, 32, and 34 to 37.

In this way, in the second embodiment, if an identification code of thepurchased commodity or the returned commodity is input via the scanner36 or the touch panel 35 and the processor 31 executes the registrationor commodity return processing based on the identification code, theprocessor 31 recognizes that an event occurs in which a commodity storedin the basket 60 changes. The processor 31 compares an image includingthe storing section 63 of the basket 60 before the event occurs, thatis, an image stored in the second memory M2 and an image including thestoring section 63 of the basket 60 after the event occur, that is, animage stored in the first image memory M1. As a result, if determiningthat a change rate between the images is larger than the secondthreshold B, the processor 31 determines that the angle of view of thecamera 38 is not blocked.

For example, a state in which a commodity 71 is stored in the storingsection 63 of the basket 60 as illustrated in FIG. 10 is represented asa first state and a state in which the commodity 71 and a commodity 72are stored in the storing section 63 as illustrated in FIG. 11 isrepresented as a second state. In this example, if the commodity 72 isstored and the first state changes to the second state, an imagephotographed by the camera 38 also changes from an image in the firststate to an image in the second state. Similarly, if the commodity 72 isremoved and the second state changes to the first state, the imagephotographed by the camera 38 also changes from the image in the secondstate to the image in the first state.

If the processor 31 recognizes that the image photographed by the camera38 changes in this way, the warning processing is not performed in theinformation terminal 30. However, if the angle of view of the camera 38is blocked, the image photographed by the camera 38 does not change evenif the first state changes to the second state or the second statechanges to the first state. At this time, the processor 31 performs thefirst warning and further performs the second warning. Therefore, in thesecond embodiment, the same action effects as the action effects in thefirst embodiment can be obtained.

The embodiments of the camera abnormality detecting device that candetect, on a real-time basis, a situation in which the angle of view ofthe camera 38 is blocked and can improve a monitoring effect by thecamera 38 are explained above. However, such embodiments is not limitedto this.

For example, in the first embodiment, the image of the upper endportions 63A, 63B, 63C, and 63D of the sidewalls 62A, 62B, 62C, and 62Dat the time when the basket 60 is viewed from above and the pair ofhandles 64A and 64B laid on the upper end portions 63A, 63B, 63C, and63D is set as the basic image P1. As another embodiment, an image of theupper end portions 63A, 63B, 63C, and 63D of the sidewalls 62A, 62B,62C, and 62D at the time when the basket 60 is viewed from above may beset as the basic image P1.

In the first or second embodiment, the second warning includes, inaddition to the processing of the first warning, the processing fortransmitting the warning signal from the wireless unit 34 to the server10. As another embodiment, the processing of the first warning isstopped and only the processing for transmitting the warning signal fromthe wireless unit 34 to the server 10 is performed. In that case aswell, the message for notifying the attendant that the angle of view ofthe camera 38 is blocked in the information terminal 30 is displayed onthe display device of the attendant terminal 40. Therefore, it ispossible to improve a situation in which the angle of view of the camera38 is blocked by the attendant.

In the first or second embodiment, as an example, the situation in whichthe angle of view of the camera 38 is blocked is detected asabnormality. As another embodiment, it is conceivable to detect, asnormality, a situation in which the angle of view of the camera 38 isnot blocked and notify the normality to the customer or the attendant.For example, in the first embodiment, if the matching rate is equal toor larger than the first threshold A in Act 12 in FIG. 7, a message fornotifying a normal state in which the angle of view of the camera 38 isnot blocked is displayed on the touch panel 35 and notified to theattendant terminal 40. Consequently, if the message is not displayed,the customer or the attendant can determine that the angle of view ofthe camera 38 is blocked.

Transfer of a computer program to the information terminal 30 can beperformed by recording the computer program in a removable recordingmedium or can be performed by communication via a network. A form of therecording medium may be any form if, like a CD-ROM, a memory card, orthe like, the recording medium can store the computer program and can beread by a device.

The several embodiments are explained above. However, the embodimentsare presented as examples and are not intended to limit the scope of theinvention. These new embodiments can be implemented in other variousforms. Various omissions, substitutions, and changes can be made withoutdeparting from the spirit of the invention. These embodiments andmodifications of the embodiments are included in the scope and the gistof the invention and included in the inventions described in claims andthe scope of equivalents of the inventions.

What is claimed is:
 1. A camera abnormality detecting device comprising:an acquiring component configured to acquire, from an image photographedby a camera provided in a conveyance body for photographing an inside ofa basket placed on a basket receiving section of the conveyance body, animage of a region where an upper edge of the basket is surmised to bereflected; and a notifying component configured to notify, based on theimage acquired by the acquiring section, whether an angle of view of thecamera is blocked, wherein the notifying component notifies, accordingto a matching rate of the image acquired by the acquiring component anda basic image showing the upper edge of the basket, whether the angle ofview of the camera is blocked.
 2. The device according to claim 1,wherein the notifying component notifies that the angle of view of thecamera is blocked if the matching rate of the image acquired by theacquiring component and the basic image showing the upper edge of thebasket is smaller than a first threshold and adds or changes anotification destination if a state in which the matching rate issmaller than the first threshold continues for a fixed time or more. 3.The device according to claim 1, wherein the notifying component isfurther configured to notify a possible remedy if the angle of view ofthe camera is blocked.
 4. The device according to claim 1, wherein thenotifying component notifies using a sound signal.
 5. The deviceaccording to claim 1, wherein the notifying component notifies using alight signal.
 6. A POS system comprising the device according toclaim
 1. 7. A camera abnormality detecting device comprising: anacquiring component configured to acquire, from an image photographed bya camera provided in a conveyance body for photographing an inside of abasket placed on a basket receiving section of the conveyance body, animage of a region where an inside of the basket is surmised to bereflected; a recognizing component configured to recognize an event inwhich a commodity stored in the basket changes; and a notifyingcomponent configured to notify, from a degree of a change of imagesacquired by the acquiring component before and after the event occurs,whether an angle of view of the camera is blocked, wherein the notifyingcomponent notifies that the angle of view of the camera is blocked ifthe degree of the change of the image is larger than a second thresholdand adds or changes a notification destination if a state in which thedegree of the change is larger than the second threshold continues for afixed time or more.
 8. The device according to claim 7, wherein thenotifying component is further configured to notify a possible remedy ifthe angle of view of the camera is blocked.
 9. The device according toclaim 7, wherein the notifying component notifies using a sound signal.10. The device according to claim 7, wherein the notifying componentnotifies using a light signal.
 11. A POS system comprising the deviceaccording to claim
 7. 12. A camera abnormality detecting methodcomprising: acquiring, from an image photographed by a camera providedin a conveyance body for photographing an inside of a basket placed on abasket receiving section of the conveyance body, an image of a regionwhere an upper edge of the basket is surmised to be reflected; andnotifying, based on the acquired image, whether an angle of view of thecamera is blocked, wherein notifying is according to a matching rate ofthe image acquired by the acquiring component and a basic image showingthe upper edge of the basket, whether the angle of view of the camera isblocked.
 13. The method according to claim 12, wherein the notifying isexecuted when the angle of view of the camera is blocked if the matchingrate of the image acquired by the acquiring component and the basicimage showing the upper edge of the basket is smaller than a firstthreshold and adds or changes a notification destination if a state inwhich the matching rate is smaller than the first threshold continuesfor a fixed time or more.
 14. The method according to claim 12, furthercomprising notifying a possible remedy if the angle of view of thecamera is blocked.
 15. The method according to claim 12, whereinnotifying comprises using a sound signal.
 16. The method according toclaim 12, wherein notifying comprises using a light signal.
 17. A cameraabnormality detecting device comprising: an acquiring componentconfigured to acquire, from an image photographed by a camera providedin a conveyance body for photographing an inside of a basket placed on abasket receiving section of the conveyance body, an image of a regionwhere an upper edge of the basket is surmised to be reflected; and anotifying component configured to notify, based on the image acquired bythe acquiring section, whether an angle of view of the camera isblocked, wherein the notifying component is further configured to notifya possible remedy if the angle of view of the camera is blocked.
 18. Thedevice according to claim 17, wherein the notifying component notifies,according to a matching rate of the image acquired by the acquiringcomponent and a basic image showing the upper edge of the basket,whether the angle of view of the camera is blocked.
 19. The deviceaccording to claim 17, wherein the notifying component notifies using asound signal.
 20. The device according to claim 17, wherein thenotifying component notifies using a light signal.
 21. A POS systemcomprising the device according to claim 17.